Recurrent Neural Networks in NLP

Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]] Part of the page on [[Artificial Intelligence/Introduction to AI/Week 2 - Introduction/Natural Language Processing]] See [[RNNS - Recurrent neural networks]] for more detail.

As words move through the neurons, the next batch of inputs enter the network in such a way that late-stage words form part of the processing of early-stage words.

This ensures context is taken into account when processing language.

Traditional RNNs are trained through back-propagation through time (BPTT).

recurrent neural network

Go back to the [[AI Glossary]]

#seq

A neural network that is intentionally run multiple times, where parts of each run feed into the next run. Specifically, hidden layers from the previous run provide part of the input to the same hidden layer in the next run. Recurrent neural networks are particularly useful for evaluating sequences, so that the hidden layers can learn from previous runs of the neural network on earlier parts of the sequence.

For example, the following figure shows a recurrent neural network that runs four times. Notice that the values learned in the hidden layers from the first run become part of the input to the same hidden layers in the second run. Similarly, the values learned in the hidden layer on the second run become part of the input to the same hidden layer in the third run. In this way, the recurrent neural network gradually trains and predicts the meaning of the entire sequence rather than just the meaning of individual words.